Hard coded the number of forked threads to 10. Should that be a parameter?

Can we do similar to sync, expose a --fork with a default

Why the per-file concurrency on top of the per-chunk concurrency? Not sure if I missed a conversation somewhere, but I suspect this might actually be slower - increasing the concurrency by a factor of ten would make us more likely to hit rate-limiting.

@olorin This is just a WIP. I think we probably need to collect file sizes as well and if the file sizes are small enough using multiple downloadSingle and for large files, use a single multipart download.

This is just a WIP. I think we probably need to collect file sizes as well and if the file sizes are small enough using multiple downloadSingle and for large files, use a single multipart download.

That sounds good. Would be good to have some empirical basis for whatever the split ends up being too.

Might make sense to fix recursiveUpload first. More obvious how to do that.

As a first pass it would be fine to have a default of 1 with an option to override as well. That way we can start using it even if its not the most efficient at the moment

When this does the right/smart thing, would we still want an option to specify the parallelism? If not it seems a bit weird to add it for now only to remove it later.